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Introduction 


There  is  growing  interest  in  the  healthcare  field  for  telemedicine,  the  use  of  teleconferencing 
technologies  (bidirectional  video  and  audio)  to  allow  consultations  between  doctor  and  patient 
located  in  different  places.  Telemedicine  reduces  travel  times  for  physicians  and  patients,  and 
allows  specialized  healthcare  to  reach  locations  that  might  otherwise  be  underserved.  With  the 
growing  acceptance  of  telemedicine  comes  the  possibility  of  other  t5rpes  of  remote  medical 
intervention,  such  as  remote  surgery.  Remote  surgery,  where  one  or  more  members  of  the 
surgical  team  are  remotely  located,  is  presently  in  its  infancy.  One  of  the  prime  motivators  of 
this  concept  has  been  the  United  States  Department  of  Defense,  who  saw  the  potential  for  far- 
forward  battlefield  medical  intervention  without  placing  highly  valuable  surgeons  in  danger. 
Presently,  implementations  consist  of  research  systems  in  which  a  remote  surgeon  instructs 
(teleoperates)  a  robot  for  suturing  or  for  micro-inspection. 

We  have  constructed  the  Integrated  Remote  Neurosurgical  System  (IRNS),  a  remotely-operated 
neurosurgical  microscope  with  high-speed  communications  and  a  surgeon-accessible  user 
interface.  The  IRNS  allows  high  quality  bidirectional  mentoring  in  the  neurosurgical  suite, 
enabling  a  remote  neurosurgeon  to  lend  expertise  to  the  Operating  Room-based  (OR) 
neurosurgical  team. 

The  IRNS  uses  a  generic  microscope/transport  model,  allowing  the  system  to  be  useful  for 
remote  operation  of  a  general  class  of  operating  room  robots  and  video  sources.  In  our  case,  the 
equipment  is  the  Carl  Zeiss  [1]  MKM  surgical  microscope/transport,  with  attached  Zeiss  OPMI 
stereo  microscope  (Figure  1).  SuMIT  (the  Surgical  Manipulator  Interface  Translator)  performs 
translation  from  generic  commands  to  hardware-specific  transport  commands  in  the  OR.  A 
SuMIT  interface  is  planned  for  each  device  to  be  controlled  by  the  IRNS. 

To  the  remote  surgeon,  the  most  important  aspect  of  the  IRNS  is  the  feeling  of  being  present  in 
the  OR  during  the  surgery.  It  is  this  sensation  of  telepresence  that  enables  the  remote  surgeon  to 
comfortably  and  confidently  control  the  remote  microscope  and  interact  smoothly  with  the  OR 
team.  To  be  successful,  telepresence  systems  must  place  a  low  cognitive  load  on  the  user  by 
allowing  him  to  interact  with  the  system  in  a  natural  and  familiar  manner.  For  example,  the  use 
of  bidirectional  audio  allows  the  users  of  a  telepresence  system  to  communicate  verbally  as  they 
would  in  person.  The  addition  of  bidirectional  video  adds  the  information  contained  in  facial 
expressions  and  gestures.  This  type  of  feedback  is  knovm  to  improve  the  interactivity  of 
collaborative  systems[2].  If  other  physical  artifacts  are  present  in  the  system,  such  as  input 
devices  with  which  the  user  interacts,  they  should  be  familiar  in  appearance  and  operation  to 
those  used  under  normal  circumstances.  In  this  manner  our  telepresence  system  can  be  useful 
and  acceptable  to  the  surgeon.  As  we  will  show,  we  have  placed  a  strong  emphasis  on 
telepresence  in  the  IRNS. 

The  IRNS  remote  workstation  incorporates  image-based  surgical  planning  capabilities  and  can 
display  real-time  video  from  both  the  microscope  and  an  overhead  video  camera.  The  overhead 
camera  shows  the  remote  surgeon  the  general  layout  of  the  OR  and  the  relative  locations  of  the 
OR  team,  the  patient,  and  the  robot.  The  IRNS  also  incorporates  the  ability  to  remotely  control 
the  microscope  position,  zoom,  and  focus.  The  remote  workstation  includes  several  methods  for 
microscope  control.  The  primary  interaction  method  is  a  force-reflecting  handgrip  mounted  on  a 
6-degree-of-freedom  robot  that  allows  the  remote  surgeon  to  intuitively  position  the  microscope. 
Other  features  include  bidirectional  audio,  image  archiving,  voice  recognition,  bidirectional 
whiteboarding,  and  an  interface  to  our  presurgical  stereotactic  planning  system. 


Real-time  simulation  of  the  microscope  transport  is  provided  by  the  commercially  available 
Telegrip  (Envision)  simulation  package  from  Deneb  Robotics[3i.  Telegrip /Envision  is  used  to 
produce  a  realistic  view  of  the  OR  robot  as  it  is  controlled  from  the  remote  workstation.  The 
simulation  permits  pre-surgical  simulation  and  training  for  surgeons  without  access  to  an  actual 
microscope  transport  system. 

The  components  of  the  IRNS  are  integrated  using  Asynchronous  Transfer  Mode  (ATM)[4] 
switching  to  provide  low  latency  data  transfer.  A  registration  system  has  been  developed  to 
ensure  precise  monitoring  of  the  patient  and  microscope. 


Figure  1.  The  MKM  microscope  transport 
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The  IRNS  provides  an  opportunity  to  assess  the  benefits  of  remote  surgical  mentoring  and  to 
gain  technical  expertise  in  teleoperation  of  surgical  devices.  By  integrating  high  performance 
workstations  and  microscope  carriers,  the  system  demonstrates  the  usefulness  of 
communications  and  computing  resources  in  improving  access  to  specialized  surgical  facilities. 
This  research,  along  with  the  more  sophisticated  systems  that  will  follow,  serves  as  a  foundation 
and  testing  platform  for  extending  the  surgeon's  skills  without  regard  to  time  zone  or 
geographic  boundaries. 

This  document  will  describe  the  IRNS  in  its  current  state,  including  the  system  architecture, 
communications  architecture,  safety  guidelines,  user  interface,  and  hardware.  We  will  then 
discuss  the  future  research  and  development  goals  for  the  IRNS. 


IRNS  System  Architecture 


Oven/iew 

The  IRNS  consists  of  two  worksites:  the  operating  room  and  the  remote  workstation.  They  are 
connected  via  an  ATM  network  (Figure  2).  Each  worksite  hosts  elements  such  as  workstations 
and  cameras.  Locally,  these  elements  are  interconnected  by  ATM,  Ethernet,  or  RS-232 
connections,  and  are  ultimately  linked  by  a  site  workstation.  The  ATM  link  is  responsible  for 
carrying  all  video,  audio,  and  control  data.  Currently,  the  remote  workstation  may  control  either 


Figure  2.  Block  diagram  of  the  Integrated  Remote  Neurosurgical  System 


a  simulated  MKM  (in  Telegrip/Envision),  a  simulated  Robotics  Research  (RR)  1607,  an  actual 
MKM  or  an  actual  RR1607.  The  MKM  has  been  installed  in  the  University  of  Virginia  Health 
Sciences  Center  Hospital  and  in  the  first  floor  of  the  Virginia  Neurological  Institute  (VNI).  The 
RR1607  has  been  permanently  mounted  in  an  OR  mockup  within  the  Neurovisualization  Lab  on 
the  third  floor  of  the  VNI  building.  The  simulated  MKM  and  RR1607  were  used  to  test  various 
components  of  the  interface  when  either  the  actual  robot  was  not  available  or  when  the  testing 
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being  done  was  potentially  dangerous  to  the  robot.  These  fully  functional  models,  complete 
with  forward  and  inverse  kinematic,  were  also  used  to  show  the  remote  operator  the  current 
configuration  of  the  actual  robotic  scope.  The  RR1607  is  interfaced  to  a  servo-controlled  charge- 
coupled  device  (CCD)  camera  that  is  analogous  to  the  OPMI  stereo  microscope.  In  conjunction 
with  technical  representatives  of  Carl  Zeiss  Inc.,  the  project  team  developed  a  teleoperation 
interface  to  the  MKM  and  integrated  into  the  IRNS.  AU  software  and  hardware  shown  in  Figure 
2,  including  the  ATM  network,  room  camera  software,  audio  communications,  and  surgical 
planning  user  interface,  have  also  been  integrated  into  the  IRNS. 

In  order  to  familiarize  the  reader  with  the  IRNS,  it  is  illustrative  to  present  a  simple  scenario  for 
its  usage.  A  typical  scenario  for  use  of  the  IRNS  consists  of: 

•  microscope  setup  and  positioning, 

•  session  startup  and  connection  to  remote  workstation, 

•  preliminary  consultation  between  OR  and  remote  physicians, 

•  commencement  of  surgery  with  remote  physician  observing  and  providing 
expertise, 

•  handoff  of  control  of  microscope  between  OR  and  remote  physicians, 

•  remote  control  of  microscope,  including  gross  and  fine  motion,  and 

•  session  shutdown. 

Each  task  in  this  list  has  now  been  developed  in  the  IRNS.  We  will  now  examine  each  item  in 
greater  detail. 

Microscope  setup  and  positioning  involves  both  microscope-specific  tasks  (such  as  driving  the 
MKM  transport  into  the  OR  and  placing  in  position)  and  IRNS-specific  tasks  such  as  establishing 
the  communications  link  between  the  microscope  and  the  OR  workstation.  Microscope-specific 
tasks  are  performed  first.  For  the  MKM,  setup  includes  physical  positioning  of  the  transport 
and  the  workstation  in  the  OR,  making  the  appropriate  cabling  connections,  booting  of  the 
MKM  workstation,  and  executing  the  MKM  control  software,  and  executing  the  MKM-IRNS 
link  software.  The  setup  phase  also  includes  the  establishment  of  the  registration  of  the  patient's 
position  relative  to  the  robot  (a  process  known  as  registration).  This  software  provides  ttie  IRNS 
generic  interface  to  the  MKM  controller.  Once  the  MKM-IRNS  link  software  is  running,  the 
system  is  controlled  generically  using  the  IRNS. 

The  IRNS  obtains  the  current  robot  position  and  the  patient  registration  transformation  from  the 
MKM  using  the  MKM-IRNS  link  software.  This  information  is  used  for  image-guided  control  of 
the  microscope. 

The  remote  workstation  is  connected  to  the  OR  workstation  next.  The  method  used  to  establish 
the  communications  link  will  be  described  in  later  sections;  it  is  transparent  to  the  users  of  the 
system  in  any  case.  Startup  of  the  remote  workstation  software  is  performed  simply  by 
execution  of  a  single  UNIX  command.  Once  the  software  is  running,  the  remote  surgeon  wiQ 
typically  specify  a  patient's  image  data  set  and  surgical  plan  to  load. 
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Once  the  connection  between  sites  has  been  established,  the  OR  and  remote  surgeons  may  greet 
each  other  verbally  and  visually.  Each  can  see  the  other  on  their  workstation  screens,  and  speak 
normally  as  if  in  a  face-to-face  conversation.  A  brief  discussion  may  take  place  about  the  surgical 
plan  or  specific  details  of  the  case.  Once  surgery  begins,  the  OR  surgeon  may  use  the  microscope 
normally,  repositioning  it  and  viewing  the  surgical  site  through  it. 

Handoff  of  control  to  the  remote  surgeon  is  designed  to  be  unobtrusive  and  positive.  That  is, 
both  sites  know  that  control  has  been  handed  off,  and  no  undesirable  or  unsafe  motions  ensue. 
The  handoff  is  initiated  verbally  by  the  remote  surgeon.  That  is,  he  simply  asks  for  control,  and 
the  OR  surgeon  may  agree  or  disagree  to  the  handoff.  If  it  is  agreeable  to  the  OR  surgeon,  a 
member  of  the  OR  team  must  press  and  hold  a  dead-man  switch  (a  switch  that  must  be 
continuously  held  in  order  to  complete  the  circuit)  in  the  "ON"  position  for  the  duration  of  the 
remote  control.  The  remote  surgeon  is  verbally  informed  that  he/ she  may  take  control  of  the 
system  which  is  completed  by  toggling  a  switch  on  remote  workstation  screen.  At  any  time, 
control  may  be  taken  back  by  the  OR  simply  by  releasing  the  dead-man  switch.  A  dead-man 
switch  was  chosen  over  a  toggle  switch  for  use  in  the  OR  to  reinforce  the  issue  of  the 
responsibility  and  control  of  the  patient's  safety  during  the  remote  operation  remains  with  the 
local  OR  team. 

The  IRNS  provides  the  remote  surgeon  with  the  ability  to  perform  gross  robot  motion,  fine 
robot  motion,  microscope  control,  image  retrieval,  and  image  capture  and  annotation  facilities. 
The  remote  surgeon  has  three  methods  of  controlling  the  remote  robot:  image  plane  translation, 
image  guided  motion  (using  preoperative  MRI/CT  images  registered  to  the  patient),  or 
analogical  motion  using  a  force  reflecting  input  device.  The  surgeon  may  generate  either  gross 
or  fine  motions  using  the  force  reflecting  input  device. 

Session  shutdown  is  performed  after  verbal  agreement  by  both  parties.  The  remote  surgeon  then 
toggles  a  control  on  his  workstation  indicating  that  the  connection  is  to  be  broken.  Alternatively, 
the  OR  team  can  release  the  dead-man  switch.  On  both  the  remote  and  OR  workstation  console, 
a  message  is  displayed  indicating  that  the  remote  surgeon  is  no  longer  in  control,  but  OR 
operation  is  not  affected.  Shutdown  in  the  OR  is  performed  by  exiting  the  IRNS  software  and 
shutting  down  the  microscope  in  the  normal  fashion. 


Communicatons  Design 

Asynchronous  Transfer  Mode  (ATM)  networking  is  an  emerging  communications  technology  in 
which  fixed-length  "cells"  of  data  are  transferred  at  speeds  of  over  2.3  Gbits/ sec[5].  ATM 
supports  different  but  guaranteed  quality-of-service  (QoS)  levels  depending  on  traffic  types,  and 
functions  similarly  over  local  and  wide  area  networks.  ATM  cells  can  be  switched  through  inter¬ 
exchange  carrier  networks  and  carried  over  earth  satellite  links,  making  it  ideal  for  remote 
delivery  of  critical  teleoperation  applications.  The  primary  advantage  of  ATM  is  that  it  was 
specifically  designed  for  simultaneous  data,  voice  and  video  transmission  -  precisely  the 
characteristics  required  for  telemedicine. 

We  have  implemented  a  private  ATM  network  for  carrying  digital  as  well  as  audio/video  data. 
The  network  is  composed  of  three  Newbridge  Network  36150  ATM  switches,  two  four  port 
switches  and  one  eight  port  model.  All  connections  both  between  the  switches  and  to  the 
network  end  points  are  150Mbit /sec  C)C3.  The  long  haul  connection  between  the  remote 
workstation  and  the  hospital  is  made  using  single  mode  fiber.  All  other  ATM  devices  are 
physically  connected  using  multi-mode  fiber.  Three  high  performance  UNIX  workstations  are 
attached  to  this  network,  two  in  the  lab  and  one  in  the  hospital  OR.  Each  workstation  includes 
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video  and  audio  sources  and  monitors.  These  are  connected  to  the  ATM  switches  via  coaxial 
cables,  where  they  are  digitized  using  JPEG  ATM  video  compression  boards.  This  system  is 
used  to  transport  a  combination  of  live  video,  audio  and  digital  control  data  between  the 
various  endpoints  of  the  network.  See  Figure  3. 

For  transmission  of  image  data  and  control/ feedback  information,  both  locally  and  remotely,  we 
use  TelRIP[6],  a  general  purpose  data  exchange  system.  TelRIP  uses  a  data-centered  approach  to 
modularity,  meaning  that  programs  communicate  with  one  another  by  specifying  the  types  of 
data  they  wish  to  send  or  receive.  We  have  developed  a  standardized  set  of  TelRIP  data  objects 
that  are  used  to  transmit  generic  information,  such  as  microscope  settings,  positions,  and 
localizer  positions.  By  using  TelRIP,  the  IRNS  has  been  developed  as  a  set  of  discrete  peer 
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Figure  3.  ATM  network  layout 
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applications  that  nm  largely  asynchronously.  An  advantage  is  that  our  processes  can  continue  to 
run  without  knowing  the  state  of  other  processes  on  the  network.  This  approach  also  allows  us 
to  standardize  the  interfaces  to  devices  and  controls  without  predetermining  the  exact 
implementation  of  each  interface. 

In  the  last  year  all  of  the  OC3  ATM  links  have  been  completed.  All  imaging,  video,  audio,  and 
control  data  is  past  between  the  remote  and  OR  sites  using  ATM  links. 
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OR  Worksite 


The  OR  worksite  consists  of: 

•  A  surgical  stereo  microscope  mounted  on  a  robotic  transport, 

•  One  or  more  'bird's  eye'  view  cameras  on  pan/tilt  heads, 

•  An  optical  registration  system,  and 

•  A  workstation  to  display  and  annotate  pre-surgical  and  inter-operative  medical  imagery. 

Live  video  from  both  the  stereo  microscope  and  the  OR  view  camera(s)  is  routed  through  JPEG 
compression  hardware  before  transmission  across  the  ATM  link  and  is  subsequently 
decompressed,  again  in  hardware,  at  the  remote  location.  The  OR  bird's  eye  view  cameras  are 
included  to  give  the  remote  surgeon  feedback  on  the  situation  in  the  OR. 

By  sensing  optical  targets  placed  on  the  robot  and  the  patient's  head  holder,  the  optical 
registration  system  precisely  monitors  the  robot's  position  relative  to  the  patient  and 
dynamically  provides  a  transform  from  the  end  of  the  robot  to  the  patient.  The  registration 
system  is  also  used  to  establish  a  transform  between  the  patient  in  the  OR  and  the  presurgically 
acquired  images  of  the  patient.  This  spatial  transform  is  calculated  by  first  digitizing  fiducial 
markers  on  the  patient  with  the  optical  registration  system's  pointing  device.  The  markers  show 
as  bright  spots  in  the  MRI  images  acquired  before  the  start  of  the  telementoring  session.  The 
locations  of  the  markers  on  the  patient  and  those  in  the  images  are  input  into  a  point-matching 
algorithm  that  then  provides  the  transform.  These  two  transforms  give  the  IRNS  the  capability 
to  move  the  microscope  based  on  a  position  selected  by  the  remote  physician  on  the  presurgical 
images.  For  example,  the  mentoring  surgeon  can  indicate  a  surgical  target  and  the  entry  point 
into  the  skuU  using  3D  images  and  the  microscope  will  be  moved  and  aligned  along  that 
trajectory. 

In  our  OR  mockup,  optical  registration  is  performed  using  an  Image  Guided  Technologies' 
Flashpoint  5000  [7].  The  Flashpoint  consists  of  three  cameras  in  a  rigid  frame  and  several 
dynamic  reference  frames  (DRF's)  containing  infrared  light-emitting  diodes  (LED's).  The  DRF's 
are  rigidly  attached  to  the  patient  head  frame  and  to  the  robot.  By  measuring  the  spatial 
relationships  of  the  LED's  on  the  DRF's  from  the  three  cameras,  the  Flashpoint  determines  the 
position  and  orientation  of  the  patient  DRF  with  respect  to  the  robot  DRF.  Once  this  relationship 
is  known,  a  point-to-point  registration  of  fiducials  is  performed  to  calculate  the  exact 
transformation  from  the  image  base  to  the  patient.  TTie  registration  is  performed  using  a  probe 
instrumented  in  the  same  manner  as  the  DRF's.  As  the  user  indicates  a  series  of  fiducials  on  the 
patient,  the  Flashpoint  returns  the  position  of  the  fiducial  with  respect  to  the  patient  DRF.  As 
each  point  is  located,  the  user  indicates  the  same  fiducial  in  the  image  database.  Once  this 
process  is  complete,  the  system  automatically  calculates  the  relationship  between  the  robot  base 
frame,  patient,  and  image  base. 

In  the  OR,  the  MKM  uses  a  built-in  optical  registration.  The  microscope  is  guided  to  focus  on 
each  fiducial  in  turn,  and  the  corresponding  fiducial  is  indicated  in  the  MR  or  CT  images.  After 
all  fiducials  have  been  manually  located  using  the  procedure  described  above,  the  system 
calculates  the  transform  between  the  robot  base  frame,  patient,  and  image  base. 

The  OR  workstation  can  display  views  from  the  stereo  microscope,  pre-surgical  MR  and  CT 
images,  and  3D  surgical  planning  images.  Whiteboarding  (described  below  in  the  User  Interface 
section)  is  supported,  allowing  the  OR  team  and  remote  surgeon  to  communicate  by  drawing  on 
the  video  images.  Users  in  the  OR  can  interactively  slice  through  patient  data,  magnify  images, 
and  view  a  rendered  volume  of  the  brain  using  thumbwheel-style  widgets  beneath  the 
windows.  In  addition,  verbal  communication  between  the  OR  and  the  remote  site  is  provided 
through  microphones  and  ATM-linked  audio. 
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Remote  Worksite 


The  remote  worksite  consists  of  a  graphics  workstation  providing: 

•  Microscope  control, 

•  Robotic  transport  control, 

•  Video  from  the  microscope  and  OR  view  cameras, 

•  Whiteboarding, 

•  Image  (snapshot)  archiving  and  annotation  (using  graphics  and  audio), 

•  Voice  recognition  commands, 

•  Force-reflecting  handcontroller  with  MKM  handgrip, 

•  Presurgical  image  display,  including  a  calculated  oblique  slice  orthogonal  to  the  view  axis, 

•  Three-dimensional  segmented  brain  model  display,  and 

•  Surgical  planning  facilities. 

Microscope  controls  that  are  extended  to  the  remote  workstation  include  manual  focus  and 
zoom  adjustments  as  well  as  the  automatic  focusing  control.  These  controls  can  be  activated  by 


using  verbal  commands  to  the  system's  speech  recognition  software.  Speech  Systems  Phonetic 
Engine  500[8]  or  by  pressing  the  appropriate  controls  on  the  handgrip.  See  Figure  4. 


Microscope  transport  positioning  control  can  be  performed  in  three  ways: 

•  Graphically  -  The  surgeon  can  manipulate  the  graphical  user  (GUI)  to  position  the 
microscope. 
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•  Verbally  -  The  surgeon  can  issue  commands  to  the  microscope  by  speaking. 

•  Analogically  -  The  surgeon  can  use  a  force-reflecting  handcontroller  to  position  the 
microscope. 

Facilities  on  the  layout  of  the  GUI  allow  the  surgeon  to  control  the  position  of  the  microscope  in 
image  space  or  in  relation  to  the  patient.  In  the  former  case,  the  surgeon  may  translate  the 
microscope  and  zoom  using  arrow  buttons  on  the  edges  of  the  video  window.  In  this  mode,  the 
orientation  of  the  microscope  remains  constant,  and  motion  is  performed  as  relative  translations 
in  the  "tool  frame"  of  the  robot.  The  control  buttons  can  be  activated  either  by  using  the  mouse 
or  by  pressing  the  touchscreen. 

In  addition,  speech  recognition  software  allows  the  remote  surgeon  to  control  panning  and 
zooming  of  the  microscope  simply  by  speaking  "pan  left",  "zoom  in",  etc.  Most  of  the  GUI 
functions  now  have  verbal  equivalents,  such  as  panning  the  microscope  and  taking  snapshots  of 
the  current  microscope  view.  To  issue  voice  commands,  the  surgeon  presses  a  foot  switch  and 
speaks  normally.  The  remote  workstation  includes  a  microphone  attached  to  the  face  of  the 
monitor,  but  a  headset  may  be  used  in  noisy  environments.  The  system  is  speaker  independent. 
Response  has  been  positive  from  our  primary  consulting  surgeon  and  casual  visitors  to  our  lab; 
this  technique  of  interaction  is  continually  being  enhanced  to  include  more  commands. 

To  position  the  microscope  relative  to  the  patient,  the  surgeon  indicates  a  target  point  (center  of 
interest)  on  the  three-dimensional  segmented  brain  display,  and  an  "entry  point".  The  entry 
point  is  a  point  situated  along  the  view  axis  of  the  microscope.  Where  the  entry  point  is  located 
on  the  axis  is  unimportant;  the  point  is  used  simply  to  calculate  a  unit  vector  for  calculation  of 
the  microscope  orientation.  The  final  position  of  the  microscope  is  calculated  by  multiplying  the 
unit  vector  by  a  fixed  view  distance.  The  view  distance  is  currently  a  system  parameter  set  to 
600mm;  however,  we  intend  to  make  the  view  distance  variable  based  on  surgeon  preference. 

Analogical  motion  of  the  microscope  is  accomplished  using  a  tabletop  robot  (Puma  260)  with  a 
MKM  handgrip  attached.  The  MKM  handgrip  provides  force  and  moment  readings  at  20  Hz  as 
the  surgeon  pushes,  pulls,  and  twists  the  device.  These  readings  are  fed  into  a  Pentium-based 
personal  computer  in  which  a  real-time  control  loop  moves  the  Puma  to  minimize  the  forces  and 
moments.  Thus,  the  system  remains  in  equilibrium  when  the  surgeon  is  not  holding  the 
handgrip;  when  the  surgeon  applies  forces  to  the  handgrip,  the  robot  moves  to  return  the  force 
value  to  zero.  The  relative  motion  is  scaled  and  applied  to  the  OR  robot  as  a  position  control 
command.  By  changing  the  force  control  equations,  we  can  approximate  dynamic  effects  such  as 
inertia. 

Because  the  force  control  system  uses  the  MKM  handgrip,  with  which  the  surgeon  may  already 
be  familiar,  and  emulates  a  free  floating  body,  training  time  is  minimized.  However,  we 
continue  to  fine-tune  parameters  in  the  force  controller  to  improve  the  control  stability,  avoid 
singularities  in  the  robot  kinematics,  and  limit  joint  and  cartesian  velocities.  The  low  sample  rate 
of  the  MKM  handgrip  causes  problems  for  smooth  motion  and  stability.  Force  control 
applications  are  more  typically  servoed  at  IKHz.  We  have  experimented  with  a  JR3 
force /moment  sensor  positioned  between  the  robot  and  handgrip.  This  sensor  provides  data  at 
IKHz.  It  gave  better  results  than  the  MKM  sensor,  but  it  adds  weight  and  lengfo  to  the  overall 
handgrip.  Another  difficulty  with  the  force  control  system  is  the  high  friction  present  in  the 
Puma.  IWs  is  a  highly  nonlinear  effect  that  cannot  be  reasonably  modeled.  We  have  adopted  the 
usual  approach  to  friction  by  increasing  control  gains,  which  is  adversely  affecting  system 
stability. 
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For  added  safety,  we  have  implemented  a  force  reflection  component  in  the  Puma  controller. 
Arbitrary  geometric  volumes  may  be  designed  and  issued  to  the  real-time  controller,  which  will 
restrict  the  Puma's  motion  to  remain  within  the  volume.  For  example,  a  conical  volume  may  be 
specified,  with  the  apex  located  at  the  center  of  interest  on  the  patient,  and  the  axis  coincident 
with  the  view  vector.  This  volume  will  restrict  the  microscope  to  remain  within  a  reasonable 
radius  of  the  original  view  vector.  As  the  surgeon  pushes  the  handgrip  toward  the  (invisible) 
edge  of  the  volume,  he  will  feel  no  resistance  until  the  boundary  is  encoimtered.  At  the 
boundary,  an  opposing  force  is  automatically  generated,  keeping  the  handgrip  within  the 
volume.  The  effect  is  quite  natural,  and  it  is  possible  to  'explore'  the  volume  and  determine  its 
basic  shape  without  a-priori  knowledge.  We  intend  to  improve  the  volume  design  features  so 
that  the  volume  may  be  automatically  generated  based  on  the  robot  and  patient  positions,  and 
to  display  the  volume  dynamically  in  Telegrip. 

Prior  to  the  force  control  system,  we  have  also  experimented  with  "props"[9]  real-world  objects 
with  magnetic  trackers  embedded  within.  There  are  two  props:  one  that  represents  the 
microscope  (a  stylus)  and  one  that  represents  the  patient's  head  (a  doll's  head).  By  monitoring 
the  orientation  of  these  trackers,  the  system  can  give  the  actual  microscope  the  same  orientation 
with  respect  to  the  patient.  This  interface  technique  has  been  in  use  at  our  lab  for  some  time  (on 
a  prototype  surgical  planning  tool),  and  has  received  enthusiastic  support  from  both  surgeons 
and  residents  for  its  ease  of  use.  In  this  application,  it  was  not  as  successful  due  to  the  difficulty 
in  performing  fine  motions  and  restricting  dangerous  or  impossible  robot  motion  commands. 

Feedback  from  the  OR  is  provided  in  the  form  of  video,  audio,  and  data.  A  graphical  display  of 
the  microscope's  current  view  vector  is  shown,  to  give  the  suxgeon  a  better  feel  for  the  spatial 
configuration  of  the  patient,  microscope,  and  transport.  The  remote  surgeon  is  presented  with 
live  (30fps)  video  from  the  microscope  and  the  O/R  room  camera  views.  Controls  on  the 
window  allow  the  surgeon  to  pan  the  OR  room  camera  in  4  directions  as  well  as  zoom. 

We  have  also  implemented  a  "point-click"  camera  control  interface[10].  This  technique  allows 
the  surgeon  to  simply  point  at  an  object  of  interest  in  the  room  view  and  the  camera 
automatically  centers  that  object  in  the  view.  The  surgeon  may  also  indicate  a  rectangular  region 
(a  "zoom  box")  in  the  view  and  the  system  wiQ  pan  and  zoom  to  align  the  edges  of  the  visible 
image  with  the  edges  of  the  indicated  region.  These  techniques  require  careful  calibration  of  the 
pan/zoom  characteristics  of  the  camera. 

Snapshots  of  the  view  through  the  microscope  can  be  taken  and  saved  along  with  the 
positioning  information  necessary  to  return  the  transport  to  the  snapshot  view.  This  allows  the 
remote  surgeon  to  make  comparisons  over  the  course  of  a  procedure,  to  document  the 
procedure  as  it  occurs,  or  to  use  the  snapshots  as  a  visual  aid  during  consultations  with  the  OR 
team.  To  return  the  microscope  to  the  position  from  which  snapshots  were  taken,  commands 
may  be  issued  via  the  graphical  user  interface  or  by  verbal  command.  A  separate  window 
normally  displays  the  most  recent  snapshot,  but  a  history  menu  of  thumbnail  images  lets  the 
user  select  an  older  snapshot  to  display.  Whiteboarding  is  supported  over  all  snapshots.  When  a 
snapshot  is  taken  at  either  site,  the  other  site  receives  a  copy  so  that  whiteboarding  remains  in 
synchronization. 

The  surgical  planning,  image  viewing,  and  audio  facilities  are  identical  to  those  on  the  O/R 
workstation.Figures  5  and  6  show  screen  views  of  the  remote  workstation. 
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TaskAnalysis 


Feedback  from  surgeons  and  residents  has  been  sought  throughout  the  development  of  the 
IRNS.  Observations  of  surgeons  using  robotic  microscopes  have  been  conducted  at  the 
University  of  Virginia  hospital,  and  have  also  been  reviewed  on  video  logs.  These  observations 
are  critical  in  determining  the  current  deficiencies  in  the  usability  of  the  surgical  microscope  as 
well  as  the  interaction  style  to  which  the  surgeons  have  become  accustomed.  Our  system  can 
correct  some  of  the  observed  problems  (operating  table  movement,  jerky  motion  of  the 
microscope)  because  of  the  increased  flexibility  afforded  by  its  remote  interface  and  registration 
system.  Interviews  were  also  conducted  to  determine  what  major  functions  surgeons  desire. 
Features  which  came  out  of  this  process  include  speech  recognition  capabilities,  a  simulated 
view  of  the  microscope  (through  Telegrip),  the  oblique  slice  view,  the  surgical  planner  interface 
and  the  snapshot  facility. 

In  addition  to  the  design  criteria  specified  by  the  task,  ctirrent  research  concepts  from  the  areas 
of  robotic  control  paradigms  and  human-computer  interaction  were  included  in  the  design.  The 
selection  of  input  devices  and  the  graphical  user  interface  layout  and  interaction  style  were 
influenced  in  part  by  previous  researdi[ll,  12]. 

In  the  area  of  robot  control,  we  are  working  to  exploit  control  strategies  that  are  adaptive  to 
time-varying  delays  and  guarantee  the  control  performance.  As  a  result  of  the  lessons  learned  in 
this  research  project,  our  work  in  teleoperation  continues  with  the  goal  to  develop  a  generalized 
approach  to  bilateral  controlled  teleoperators  over  an  ATM  network.  We  are  concentrating  on 
source  traffic  characterization  in  ATM  networks  and  the  design  of  bilateral  control  laws  in 
presence  of  the  time-varying  delay.  The  work  includes  the  bandwidth  scheduling,  the  estimation 
of  end-to-end  transfer  delay  bound,  and  statistic  analysis  of  the  delay  pattern.  We  discuss  this 
analysis  beginning  below. 

Robot  Simulation 

Models  of  the  RR1607  and  the  MKM  have  been  developed  in  Telegrip  so  that  the  actions  of  the 
robot  could  be  simulated  when  it  was  not  actually  available.  The  MI^  simulation  includes  a 
closed-form  inverse  kinematics  module  linked  into  Telegrip  through  a  shared  library.  The 
closed-form  solution  is  more  rapid  than  the  iterative  one  and,  more  importantly,  is  capable  of 
determining  the  singular  positions  within  the  workspace  of  the  microscope  transport.  The 
RR1607  model  is  controlled  using  either  the  built-in  Telegrip  iterative  inverse  kinematics 
solution  or  a  solution  we  have  developed.  It  is  not  possible  to  derive  a  closed  form  solution  for 
the  RR1607  because  of  the  redundancy  in  its  seven  degree-of-freedom  design.  However,  it  was 
necessary  to  develop  our  own  solution  so  it  could  be  installed  in  the  OR  mockup.  The 
simulation  models  can  be  manipulated  locally  through  the  Telegrip  UI  and  remotely  through 
any  TCP/IP  link.  Various  other  components  of  the  O/R  have  also  been  modeled  in  this 
simulation  including  the  camera,  optical  tracker  and  patient. 


Issues  in  Bilateral  Control  of  Teleoperator  over  ATM  Networks 


This  section  describes  in  more  detail  our  work  on  developing  advanced  control  techniques  over 
ATM  networks.  The  work  was  critical  to  the  development  of  the  IRNS,  since  it  is  the  basis  of  our 
analogical  control  system  (the  Puma  desktop  robot).  However,  the  work  is  applicable  to  any 
teleoperator  system  built  on  ATM  and  in  this  context,  it  continues.  Teleoperators  are  often 
master-slave  systems.  The  operator,  watching  a  visual  representation  of  the  worksite,  performs 
a  task  by  moving  the  master  as  if  it  were  performing  the  task.  The  time-varying  position  control 
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inputs  of  the  master  are  sent  to  the  remote  site  as  time-varying  position  control  inputs  for  the 
slave.  The  slave  therefore  executes  a  scaled  copy  of  the  master's  motion.  When  the  slave 
contacts  the  environment,  position  error  signals  and  contact  force  information  are  generated. 
These  are  fed  back  to  the  master  and  are  used  to  backdrive  its  links,  giving  a  sense  of  contact 
with  the  remote  task.  Properties  of  the  ATM  link  determine  how  successful  this  sense  of  contact 
really  is. 

The  task  performance  is  improved  by  providing  the  contact  force  information  to  the  master. 
When  the  measured  contact  force  between  the  slave  and  the  environment  is  fed  back  to  the 
motors  of  the  master,  the  contact  force  is  said  to  be  "reflected",  and  the  teleoperator  is  said  to  be 
controlled  bilaterally[13].  The  block  diagram  of  Figure  7  represents  a  bilateral  controlled  system. 
The  operator  commands  a  velocity  forward,  through  the  master,  corrimimication  block,  and 
slave,  to  the  environment.  Likewise,  the  force  sensed  at  the  environment  is  transmitted  back 
through  these  blocks,  to  the  human  operator. 


The  research  on  bilateral  control  over  ATM  networks  is  significant  in  two  ways.  First,  the 
bilateral  control  with  time-varying  delay  remains  an  open  problem;  second,  ATM  switching  is 
an  emerging  communications  technology  that  is  well  suited  to  the  problem  of  remote 
operations.  The  primary  advantage  of  ATM  is  that  it  was  specifically  designed  for  simultaneous 
data,  voice  and  video  transmission:  precisely  the  characteristics  required  for  the  teleoperation  in 
general.  No  work  on  bilateral  control  over  ATM  has  been  published  in  the  robotics  field. 

Our  study  concentrates  on  two  aspects.  First  is  source  traffic  characterization  in  ATM  networks, 
including  characterization  of  source  traffic,  estimation  of  end-to-end  transfer  delay  bound,  and 
statistic  analysis  of  the  delay  pattern.  Second  is  design  of  the  adaptive  bilateral  control  law, 
which  includes  an  adaptive  bilateral  control  law  dealing  with  the  unknown  but  fixed  latency, 
and  an  adaptive  control  law  dealing  with  time-varying  latency  with  known  pattern.  The  stability 
of  the  bilateral  controlled  teleoperator  will  be  proved  as  well. 


Discussion  and  Conclusions 


Over  the  last  year,  the  project,  as  it  was  orginally  conceived,  has  been  completed.  There  are 
several  points  related  to  the  project's  current  status  and  the  status  of  remote  robotic  surgery  that 
warrant  comment. 

The  characterization  of  teleoperation  parameters  on  ATM  networks  will  be  an  important  effort 
for  not  only  this  project,  but  future  teleoperation  work  in  general.  We  have  one  Ph.D.  student 
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who  will  continue  her  dissertation  project  in  this  area  after  the  completion  of  the  project.  She 
and  other  members  of  the  project  have  already  published  several  papers  on  the  subject  and 
presented  them  at  national  meetings  in  the  field  of  telerobotics. 

Safety  issues  are  crucial  in  a  human/robotic  system  such  as  the  IRNS.  While  we  have  done  one 
preliminary  study  on  the  relevant  safety  issues  and  a  study  on  the  fault  tolerant  capabilities  of 
the  system,  we  have  not  yet  incorporated  sufficient  safety  precautions  in  the  IRNS.  Still  of  top 
priority  in  this  region  of  the  project  is  the  development  of  a  facile  method  for  defining  operating 
regions  of  the  scope  before  the  remote  surgeon  is  given  control  and  while  that  control  is  retained 
remotely.  Currently,  it  is  not  possible  to  automatically  determine  the  safe  operating  boundaries 
of  the  scope  in  a  dynamic  manner.  With  relative  ease,  the  system  can  determine  operating 
boimdaries  relative  to  the  patient,  given  the  optical-tracking  of  the  sensors  placed  on  the  patient. 
However,  there  is  the  additional  concern  for  other  elements  present  in  the  OR.  It  is  critic^  to 
have  this  feature  as  the  positions  of  OR  equipment  and  personnel  are  continually  changing  and 
the  safe  operating  boundaries  must  reflect  this  in  real  time.  For  each  object  or  person  to  be 
tracked  is  clearly  unreasonable  and  prohibitively  costly.  Future  advances  in  the  area  of  3D 
scene  reconstruction  from  a  limited  number  of  views  may  offer  an  acceptable  alternative.  Such 
advances,  however,  are  distant  and  certainly  remote  when  considering  the  real-time 
requirement  of  the  reconstruction. 

In  the  last  year  of  this  project,  the  team  has  concentrated  on  two  aspects  of  the  system  beyond 
the  completion  of  the  initial  requirements.  The  first  of  these  is  reducing  the  size  of  the  remote 
worksite.  In  the  working  remote  site,  there  are  at  a  minimum  of  three  computers  as  well  as  a 
robot  controller,  and  the  small  robot  hand-controller.  At  the  time  of  the  project's  inception,  the 
Intel-based  PC  platform  was  incapable  of  the  graphics  performance  necessary  for  the  surgical 
planning  aspects  of  the  project.  At  the  same  time,  the  UNIX  workstations  were  (and  continue  to 
be)  lacking  solutions  and  options  in  the  arenas  of  voice  recognition,  live  video,  and  real-time 
control.  Robotics  controllers  on  a  PC  card  were  also  unavailable.  As  a  result  of  a  mixture  of 
PCs  and  Unix  workstations  together  with  a  dedicated  controller  for  the  robot  were  necessary  to 
accomplish  the  goals  of  the  IRNS  project.  Recent  advancements  in  PC  architecture,  especially  in 
graphics  performance,  and  the  migration  of  robotics  control  systems  from  workstations  have 
indicated  that  future  efforts  on  the  project  should  be  directed  towards  the  development  of  a 
single  PC  workstation  for  the  remote  site.  Clearly,  this  PC  would  be  far-from-the-average  PC  but 
it  is  still  conceivable  that  a  modem  multiprocessor  PC  can  perform  all  the  functions  of  the  IRNS 
remote  worksite.  This  direction  has  lead  us  to  begin  porting  our  imaging  and  surgical  planning 
software  to  the  PC  platform  and  several  of  the  project  team  have  been  working  in  this  direction. 

Improvements  to  the  IRNS  GUI  will  be  greatly  facilitated  by  the  transition  to  the  PC  platform. 
Video  systems  available  on  PCs  enable  the  underlying  whiteboarding  system  of  the  IRNS  to 
overlay  graphics  onto  our  video  streams.  This  opens  the  possibility  for  more  complex 
enhancements  such  as  overlaying  of  segmented  images  and  augmented  reality  elements  [14]. 

The  second  aspect  that  we  have  we  concentrated  on  is  the  development  of  an  improved 
handcontroUer  for  the  remote  workstation.  As  will  be  discussed  later,  the  primary  acceptance 
inhibitor  of  robotic  systems  in  the  surgical  arena  has  been  the  "feel"  of  the  control.  We  have 
worked  to  improve  ttie  haptics  of  the  Puma-based  hand-controller.  The  best  solution  involved 
two  modes  of  motion;  one  for  fine  motion  and  one  for  gross  motions.  Within  each  of  these 
modes,  the  haptics  of  the  hand-controller  are  acceptable.  However,  the  mechanical  design  of  the 
Puma  limits  the  available  options  for  a  smooth  transition  between  these  modes.  Current  force 
feedback  positioning  systems  that  are  commercially  available  lack  the  number  of  degrees  of 
freedom  necessary  for  positioning  a  surgical  microscope  in  the  manner  to  which  the  surgeons 
are  accustomed.  To  address  this  issue,  we  are  reinvestigating  the  use  of  tracked  "props"  with 
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force  feedback  for  scope  positioning.  To  pursue  this  end,  the  P.L  has  also  secured  a  grant  from 
the  industrial  sector. 

Perhaps  the  most  distressing  component  of  the  project  has  been  the  lack  of  acceptance  of 
robotics  in  the  surgical  field.  When  introduced  in  1994,  microscopes  with  robotic  transports 
were  thought  to  be  the  future  of  the  precision  surgery.  However,  this  prediction  remains 
unfulfilled.  The  manufacturers  of  these  systems  have  yet  to  produce  a  robotic  microscope  with 
the  ease  of  positioning  afforded  by  the  current  standard  of  counter-balanced  scopes. 
Additionally,  the  robotic  scopes  together  with  there  necessary  image-navigation  systems  have 
proven  very  time  consuming  to  setup  and  this  has  only  contributed  to  their  lack  of  acceptance. 
The  future  of  robotic  surgical  microscopes  thus  remains  unclear.  This  outlook,  in  turn,  indicates 
a  dim  future  for  remote  microsurgical  mentoring.  Still,  the  research  within  this  project  has 
contributed  to  the  body  of  knowledge  of  force-feedback  control  of  robotic  surgical  devices  and 
the  research  within  this  lab  will  be  continuing  to  address  these  issues. 
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